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Book I; A Linguistic Approach to Protocol Analysis 


1. Introduction 


1.1. SPADE: A Linguistic Theory of Design 

1.2. PATN: Analysis by Synthesis 

1.3. Theoretical Interpretations 


1.1. SPAD E: A Linguistic Theory of Design 

In recent research we have developed a theory of design called SPADE 

which provides a model of the planning and debugging processes. 1 We contend that, 

in addition to being a powerful theory of machine problem solving. SPADE is also 

a useful framework for describing human problem solving. To support this 

contention, we apply the SPADE theory to the task of analyzing problem solving 
protocols. 


By adopting this methodology we follow the precedent established in 
seminal protocol analysis studies conducted at Carnegie Mellon University 
[Newell 1966; Newell & Simon 1972; Waterman & Newell 1972, 1973; Bhaskar & 
Simon 1976]. Our work extends their approach along three dimensions. 


l 


2. 


With the exception of the recent Bhaskar & Simon effort, the CMU 

been restricted to very limited domains such as 

thp P rtni th r iC ' Rather than limiting the task domain, we limit 
.[ff of responses. Typically protocols are transcriptions 

iIt!J 1 a t 'V al0Ud verbalizations; we focus on the more restricted 
interactions arising from a problem solving session at a 

iSrlt C0ns0le : . The analysis task in this setting is to 
interpret user actions — editing, executing, tracing, etc. — 

in terms of the SPADE theory of planning and debugging. 

The CMU theory centers on the production systems model Althouah 
productions are Turing universal, they tend to result in a JeS 

of r t C he r sP»n r F° 9 ^ organl2atlon than the linguistic formalisms 
f *j 1 ® s PADE theory. The PATN program, the procedural 

“Tu* °/ ? 0 ha SPADE th80ry ’ “ ses an tr«.s "tl*i 

network [Woods 1970] to represent planning knowledge. 
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3. CHU analyses are based on the problem behauior graph. Pursuina 

l!!te a rn?etTion t Vr PUt r aU , 0nal K lln9ulstlcs ' we deflne a " 
L * on of a Protocol to be a parse tree supplemented by 

r cn n,Ht, P T! iC annotation. The parse tree characterizes 

nS A ructure of the protocol. Semantic and 

d-h ann ° tatl0n variables an ^ assertions attached to 

[ ra °. n h ® parse tree - formalize the problem description and 

2®r f °r particular planning choices. Annotated parse 

ees closely reflect the local structure of PATN's linauistic 

regardina individ l h* rr ^ i^g more directly to inferences 
behavior graphs differences than is evident from problem 


Ruven Brooks [1975] applied the CMU approach to the programming domain, 
developing a model of coding - the translation of high level plans into the 
statements of a particular programming language - and testing the model by 
analyzing protocols. His model is a set of production rules whose conditions 
match the patterns of plan elements and whose actions generate code statements. 
Protocols are analyzed manually, with the experimenter attempting to infer the 
plan which is then expanded by the production system into code paralleling that 
of the protocol. The processes of understanding the problem, generating the 
Plan, and debugging are not formalized. SPADE goes beyond this in that it can be 
used to parse protocols and that the parse constitutes a formal hypothesis 

regarding not only the coding knowledge but also the planning and debugging 
strategies employed by the problem solver. 

The paper is divided into two books. Book I develops SPADE's linguistic 
paradigm for protocol analysis. A prototypical elementary programming protocol 
is parsed, and the implications of this information processing analysis for 
constructing cognitive models and designing computerized tutors are discussed. 

Book I does not address the question of how a protocol parse is derived. 
In earlier work, problem solving protocols were analyzed manually. 3 However, 
manual analysis is tedious and informal; hence Book II presents the design for 
PAZATN, an automatic protocol analyzer. PAZATN uses PATN 4 - the procedural 
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embodiment of the SPADE theory ~ as a generator for possible interpretations of 

the protocol, with bottom-up evidence biasing PATN toward plans which are likely 
to match the data (figure 1:1). 

PAZATN is a domain independent framework for constructing specialized 
protocol analyzers. To apply PAZATN to a particular task domain, euent 
specialists (ESP's) are supplied which embody domain-specific knowledge. For 
concreteness, we employ examples from the Logo elementary graphics programming 
domain; ESP's for this domain are discussed. PAZATK's operation is hand- 
simulated on an elementary protocol from this domain. 

1.2. PA TN: Analysis by Synthesis 

A major insight of the generative grammarians (e.g., Chomsky [1965]) was 

that it is often helpful to characterize phenomena synthetically: one devises 

rules to generate the phenomena. Analysis can then be viewed as a recognition 

process for selecting derivations from the space of synthetic possibilities. We 

adopt this viewpoint in analyzing protocols, with PATN as our generative 
formalism. 

The SPADE theory, which PATN embodies, begins with a taxonomy of commonly 
observed planning techniques (figure 1:2). When a problem is confronted — 
according to the theory - one of three types of plans may be pursued. (1) The 
problem may be solved by identification: recognizing it as already having a 
solution. This planning category, seemingly trivial, is of course essential to 
avoid infinite regress. ( 2 ) The problem may be solved by decompost lion : 
dividing it into smaller, easier subproblems. These are each solved separately, 
and then recombined, thereby disposing of the original problem. (3) Should the 
first two strategies fail, the problem may be solved by reformulation: 
redescribing it in terms that seem more amenable to solution. The reformulated 
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FIGURE Iil TOP LEVEL ORGANIZATION OF THE PROTOCOL ANALYZER 
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problem must, in turn, be solved by identification, decomposition, or further 

reformulation. As the figure indicates, each of these categories of plans is 
further subdivided. 

PATN (figure 1:3) is a problem solving program based on this taxonomy. 

PATN was derived by first representing the taxonomy as a recursive transition 

network. 5 This produces a non-deterministic problem solver. Supplying precedence 

ordering for arcs from each node, predicates which test preconditions for 

transitions, and actions to be performed when arc transitions occur, produces an 

augmented transition network [Woods 1970] that is far more deterministic in 

solving problems (although backup is permitted). The predicates access registers 

which store semantic information about the problem; the actions modify these 
registers. 


PATN-s solutions can exhibit rational bugs ~ errors arising from 
heuristically justifiable but incorrect planning decisions - such as the trial 
execution of an incomplete plan, which omits necessary interface steps. Hence a 
complementary theory of debugging is developed using the same approach as in the 
planning theory. Figure 1:4 shows a taxonomy of debugging techniques. This 
taxonomy bifurcates into techniques for diagnosing the underlying cause of a bug 
and techniques for repairing the bug once isolated, fiodel diagnosis is typical 
of the diagnostic techniques. It consists of executing the program in order to 
construct a list of violated model predicates, which is then examined to check if 
any code was written to accomplish the violated predicates. As in the planning 
theory, the debugging taxonomy is transformed into an ATN (figure 1:5) by 
providing registers, arc predicates and so on. The debugging ATN is called DAPR 
(debugger of annotated programs), and is an integral part of the PATN system. 

Consider the operation of the planning ATN on an example from the Logo 
graphics environment, where students define, test, and debug procedures to draw 
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FIGURE 1:3 A SIMPLIFIED VIEW OF PATN 
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FIGURE 1*4 A TAXONOMY OF DEBUGGING TECHNIQUES 


DEBUG 


DIAGNOSES 


PARSE-ADVISE (planning choices) 


■CODE 


PRINTOUT 


-ADVISE (rational form violations) 


MODEL-ADVISE (model violations) 


PROCESS 


ASK 


TRACE 


DO 


REPAIR 


COMPLETE 


CORRECT 



(EXISTS (MODEL-STATEMENT STEP LOC) 

(AND (MEMBER (NEGATE MODEL-STATEMENT)(:VIOLATIONS LOC)) 
(EQUAL MODEL-STATEMENT (:MODEL STEP)) 

(MISSJNG~SIEP (:PLAN LOC)))) _ 
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FIGURE 1:5 DAPR: PATN'S DEBUGGING ATN 
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simple pictures. The WISHINGWELL (figure 1:6) is a typical beginner's project. 
The students' task description is a sketch of the desired picture. PATN, 
however, requires a formal task description: figure 1:7 illustrates this 
description, the model, for WISHINGWELL. Models are expressed in an assertional 
formalism developed by Goldstein [1974, 1975], which is similar to the first 

order predicate calculus. The model characterizes the range of pictures which 
match the sketch. 6 

PATN's solution to the WISHINGWELL task has three aspects: (1) an 
hierarchical plan derivation, summarizing the arc transitions which were 
followed; (2) a snapshot of the values of the ATN's registers attached to each 
node of the derivation, representing the semantic context at the time the node 
was created; and (3) a set of instantiated arc predicates at each node 
describing why the chosen arc transition was preferred to its competitors; these 
are called the pragmatic assertions of the node.^ The semantic variables and 
pragmatic assertions relate the subgoal structure of the problem solving protocol 
to the model describing the task to be accomplished. 

Figure 1:8 shows PATN's hierarchically annotated solution. Naturally, 
this is not the only solution to the WISHINGWELL problem: to apply PATN to 
protocol analysis, we allow PAZATN to reject solutions which do not match the 

protocol data, forcing PATN to backup so that alternative solutions are 
generated. 

1.3. Theoretical Interpretations 

We define an interpretation of a protocol to be a PATN plan derivation: 
a parse tree whose fringe is the list of events (e.g., figure 1:8), augmented by 
annotation associated with each node of the parse. Since different plans 
sometimes lead to the same coding events, some protocols have more than a single 
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FIGURE 1:6 
WISHINGWELL PICTURE 
AN ELEMENTARY LOGO GRAPHICS PROJECT 
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Draw a WISHINGWELL with a square well and a triangular roof, 
connected by a pole which is a line. The roof should be above 
the pole, and the pole above the well. The well should be 
connected to the pole at the midpoint of the upper side of the 
well and the lower endpoint of the pole. The pole should be 
connected to the roof at the midpoint of the bottom side of the 
roof and the upper endpoint of the pole. The bottom side of the 
roof and the upper side of the well should be horizontal. 


(DEFINE-MODEL WISHINGWELL () 

(EXISTS (ROOF POLE WELL) 

(AND (TRIANGLE ROOF) 

(LINE POLE) 

(SQUARE WELL) 

(ABOVE ROOF POLE) 

(ABOVE POLE WELL) 

(EXISTS (P) 

(AND (CONNECTED WELL POLE (AT P)) 

(EQ P (MIDDLE (UPPER (SIDE WELL)))) 
(EQ P (BOTTOM (ENDPOINT POLE))))) 

(EXISTS (Q) 

(AND (CONNECTED POLE ROOF (AT Q)) 

(EQ Q (MIDDLE (BOTTOM (SIDE ROOF)))) 
(EQ Q (UPPER (ENDPOINT POLE))))) 
(HORIZONTAL (BOTTOM (SIDE ROOF))) 

(HORIZONTAL (UPPER (SIDE WELL)))))) 


Figure 1:7. Predicate Model for a Wishingwell 
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interpretation. The basic claim of this paper is that PATN can efficiently 
generate the psychologically plausible interpretations. 

Evidence for this claim rests on four sources of evidence. 

1. The heuristic adequacy of PATN as a problem solving program 
provides suggestive, though by no means decisive, evidence. At 
least for the restricted world of elementary Logo graphics, 
hand-simulations indicate that PATN is heuristically adequate. 

Z. Introspection by human problem solvers is a weak but useful 
source of evidence. To some extent PATN was designed on the 

basis of introspection and hence has some support along this 
dimension. 

3. A strong source of evidence is the appropriateness of the replies 
of a question-answering module that performs retrievals and 
simple inferences over a database composed of these 
interpretations. The question-answering module is introduced in 
chapter two. The replies to the example questions given in that 
chapter seem appropriate to the authors. 

4. The strongest source of evidence is ability to predict 
performance in future situations on the basis of past behavior. 

Chapter three describes modifications to the ATN that provide 
predictive models of typical problem solving behaviors. 


We find this informal evidence sufficiently encouraging (as detailed in 

the remainder of Book I) to warrant the design (in Book II) of a precise 

framework for generating SPADE-style protocol interpretations. Future research 

will rigorously evaluate the psychological validity of these interpretations as 
follows. 

1. PATN will be implemented and tested on a broad range of examples. 

This will confirm its heuristic adequacy. 

2. An editor based on SPADE will be constructed as a structured 
programming environment, and transcripts of the problem solving 
behavior of programmers using this editor analyzed. Coupled 
with systematic interviews, this will provide evidence regarding 

the sufficiency of SPADE's repertoire of planning and debugging 
concepts. 
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3. PAZATN with a question-answering interface will be implemented. 

The appropriateness of the replies generated by the question¬ 
answering module will be judged by skilled but unbiased 
informants, and by systematic subject interviews. 

4. A modeling component will be implemented that modifies the PATN 
ATN to be more in accord with a particular student's behavior. 

Tests will be conducted to determine whether the modified ATN is 

more successful in predicting performance on subsequent 
protocols. 

Before proceeding, a possible misconception involving the distinction 
between representational frameworks and psychological theories should be 
dispelled. Two hypotheses are defended by the research program outlined in this 
paper: (1) that ATN's are a useful representation for the models we are 

developing; and (2) that particular ATN's, the output of our modeling procedure, 
constitute theories of individuals -- stated in the language of ATN's -- which 
make statements about the presence or absence of certain problem solving skills. 
Both hypotheses are of course subject to experimental verification. We do not 
argue that other Turing-universal formalisms (such as productions or Heidorn's 
[1975] augmented phrase structure grammars) cannot also represent these theories. 
Stronger claims regarding the validity of ATN's per se as psychological 
mechanisms require additional assumptions regarding processing costs and 
limitations which we are not currently prepared to defend. 
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2. An Example of Protocol Analysis as Parsing 

2.1. An Example Problem Solving Protocol 

2.2. Structural Description 

2.3. Semantics 

2.4. Pragmatics 

An analogy to computational linguistics has been fruitful both in 
defining the objectives of analysis and in designing the PAZATN system for 
automating the analysis process. The analogy suggests partitioning analyses into 
syntactic, semantic, and pragmatic components. These components correspond to 
the potential control paths, data flow, and branching conditions of a procedural 
problem solver. From a problem solving standpoint, these are modeled by the 
network of states and arcs, the registers, and the transition conditions of the 
augmented transition network. From a protocol analysis standpoint, syntax is 
represented as a parse tree; semantics and pragmatics are represented as 

annotation (variables and assertions) associated with each node of the parse 
tree. 

This chapter presents a SPADE interpretation for a typical WISHINGWELL 
protocol. Book II provides a hand-simulation of PAZATN generating this 
interpretation. 

2.1. An Example Problem Solving Protocol 

Since analysis consists of the selection of a PATN plan derivation, 
analyzing a protocol identical to PATN's default solution (figure 1:8) is 
trivial. Hence, a different protocol, involving a variety of plans including 
reformulation and repetition, serves as our example. 8 
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The student begins by writing an iterative procedure to draw the 
square WELL. 

E01 ?TO WELL 

E02 >10 REPEAT 4 [20 30] 

E03 >20 FORWARD 100 

E04 >30 RIGHT 90 

E05 >END 

A superprocedure for the WISHINGWELL is defined by a sequential 
plan drawing first TREE, a previously defined procedure, and 
then WELL. 

E06 ?TO WW 

E07 >10 TREE 

E08 >20 WELL 

E09 >END 

The WW program is executed, producing figure 1:9. 

E10 ?WW 

The program is edited to include an interface establishing the 
proper relation between TREE and WELL. 

Ell 7EDIT WW 

E12 >13 RIGHT 90 

E13 >15 FORWARD 50 

E14 >17 RIGHT 180 

E15 >END 


2.2. Structural Description 

The result of analyzing this protocol is a data structure, the 
interpretation, consisting of syntactic, semantic, and pragmatic components. The 
syntactic component, diagrammed in figure 1:10, is the protocol's structural 

description: a parse tree representing the sequence of PATN arc transitions 
required to generate it. 9 

Such structural descriptions capture one aspect of problem solving 
behavior. They provide a formal basis for answering questions regarding which 
plan types were used, a topic which could otherwise be discussed only 
intuitively. 10 Their most direct application is to answering "how questions.” 
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TURTLE BEGINS HERE 
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TURTLE ENDS HERE 
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FIGURE 1:9 WW AT E10 — INTERFACE NEEDED 
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Ql. How does procedure WELL accomplish a square? 

Al. WELL uses a repetition plan. The generic subgoal is a side. 


Q2. How does procedure TREE accomplish a roof and pole? 

A2 ‘ I?? *? d po J e model parts were regrouped into a tree by a 

li^rlrv all ° n / n * Procedure TREE was already in the answer 
library, allowing an identification plan* 


Still, the parse tree is an incomplete description: it does not Indicate 
the semantic relationships between subgoals or the pragmatic criteria governing 
the choice of one plan over another. 

2.3. Semantics 

Semantic annotation is defined to be the values of semantic variables 
associated with each node of the parse. These variables relate the plan to the 
formal problem model by recording the contents of the ATN's registers at the time 
the node was generated. The following are typical PATN registers. 


1 . 


3. 


4. 


nirrlnt^ th H e hierarch ically annotated program defined below the 

d s’ reflectin 9 it; s state after dominated editing 
events have been processed. a 


2. :CODE is the fringe of :PLAN. 


ripf^noH d ^ cription of ^e effect of executing the code 

rcfi?. d bel °! f the current node. Since the code may contain 
references to undefined user procedures, rEFFECT may be 

unassigned at a given node. For the elementary graphics domain 

“ » «;io« :PICTURE, and describes the pictu"; 

drawn by the program in Cartesian coordinates. 

aJrnmin^h the r S8t ° f predicates which :C0DE is intended to 
"MODEL. 11 h * F 3 C ° rreCt progran sEFFECT is an instance of 


rADVICE is 
actions. 


a list of planning suggestions generated by PATN arc 
For example, the linearization arc (see PATN's 
conjunction node in figure 1:3) creates advice regarding both 
the order in which subprocedures should be written, and the 
order in which they should be invoked. tn 
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6 . 


is a list of warnings for potential bugs generated by 

examnlp 16 ^ f heuristic guidelines are used in planning For 

lf 00 iateractlons ar e detected when solving a problem 
thl Ir 1 !!? “I unfamiliar domain predicate, it is possible that 
SjtSr di w at ® actually give rises to interactions, but their 

•CAVEATS* ea^h n0t / et bee " learned by the system. Hence, 
t?Si nn fr n ’ recording this potential bug, on the arc 
transition from conjunction to sequential Dlans Thi«t 

diagnosi 10n 9Uid ® S DAPR ’ PA ™' S debugging module, in subsequent 



2 “ ia the list of model predicates which are not 
•EFFECT 6 lr« y JY :EFFECT , achieved by :CODE. This register and 
Goldstlin [1974] y 8 performance 'notation module designed by 


Let us sample the values of the semantic variables at various nodes of 
the parsed WW protocol. :MODEL for the top level SOLVE node was shown in 
figure 1:7. For the INT-TV SOLVE node, :MODEL is: 

The tree must be above the well, and the bottom endpoint of the 
tree must connect to the midpoint of the upper side of the well. 

In our LISP-oriented model language notation this is represented as: 


(AND (ABOVE TREE WELL) 

(EXISTS (P) 

(AND (CONNECTED TREE WELL (AT P)) 

(EQ P (MIDDLE (UPPER (SIDE WELL)))) 

(EQ P (BOTTOM (ENDPOINT TREE)))))). 

This submodel reflects the reformulation of WISHINGWELL into a TREE and a WELL. 

* 

Typically, semantic annotation is relevant to answering "what 
questions." The above value of .-MODEL for the INT-TV node provides an example. 


Q3. What is the purpose of lines 13, 15 and 17 of WW? 

A3 ‘ In°H Se uJu ee Hi 6 * w in ; line COde interfac ing subprocedures TREE 
and WELL. The interface establishes connectivity at the 

appropriate point, and causes the tree to appear above the well. 


•VIOLATIONS at the PLAN node for WW provides another example. 
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04 * eJei?t 1 E S loT? 0n9 WUh pr ° Cedure W when u is first executed (at 


A4. 


have not bean IIIlll 5 1 th ® model parts TREE and WELL 
s that b tho tr b \ 1Shed: s P ecificall y. there is no point P 
th 4 th * ree is connected to the well at P, P is the 

?he tree UPPer ° f the W ® U ’ and P ls the lower endpoint S? 


The .VIOLATIONS variable which mediates this answer is non-empty at the PLAN node 

because the debugging which generates the missing interface has not yet occurred: 
the English answer simply paraphrases its LISP value: 

(NOT (EXISTS (P) 

(AND (CONNECTED WELL TREE (AT P)) 

(EQ P (MIDDLE (UPPER (SIDE WELL)))) 

(EQ P (BOTTOM (ENDPOINT TREE)))))). 

% 

2.4. Pragmatics 

Pragmatic annotation is defined to be a record of the justifications for 
selecting a given arc transition over its competitors, and constitutes an 
hypothesis about the reasons for using a particular plan. REASONS are assertions 
attached to each node of the parse. The REASON for using a particular plan in a 
particular situation is an instance of the arc predicate leading to the ATN state 
for that plan, where the current values of the registers are taken into 

account." For example, the reason that WELL was decomposed using a repetition 
Plan in the protocol is that rMODEL at that node was generic. 

(REASON (REPETITION E02) 

(GENERIC (rMODEL E02))). 

Pragmatic annotation is germane to answering "why questions." 


Q5. Why did the student execute WW at event E10 -- did (s)he beliav* 
the program to be correct? ' e Delieve 

A5. Probably the student expected bugs. A reasonable strateqy is to 
initially pian only for the main steps, with the interfaces 
solved later by debugging. WW was executed at E10 in order to 
discover what interfacing, if any. was needed. 
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This Illustrates the analysis of a procedure containing the rational bug 

of constructing an incomplete plan. 12 Debugging operations are analysed by 

postulating the application of some DAPR technique. The reasons for debugging 

operations typically involve localising or repairing the cause of some model 

violation. The purpose of running the program at Eld was to perform model 

diagnosis: this technique was chosen because the occurrence of two consecutive 

mainsteps (with no explicit interface) implies that the plan may be incomplete: 

(REASON (MODEL-DIAGNOSIS E10) 

(AND (OPTIONAL (INTERFACE TREE WELL)) 

(MISSING (INTERFACE TREE WELL) 

(:PLAN E06)))). 

In this case, model diagnosis demonstrates the existence of violated predicates 

for which no code exists: the plan is in fact incomplete. This is the reason 

f0P th « subsequent editing: repair of the incomplete plan by resuming planning 
at the offending locale. 


l - th con *Pletion plan in the editing episode (E12 
through E14) is to eliminate the violations by supplying the 
missing interface between THEE and WELL. applying the 


(REASON (COMPLETE (E12 E13 E14)) 
(AND 
(MEMBER 
'(NOT 

(EXISTS (P) 


(AND (CONNECTED WELL POLE (AT P)) 

, - ))) 

(VIOLATIONS E10)) 

(EQUAL 

'(EXISTS (P) 


(AND (CONNECTED WELL POLE (AT P)) 

, " » 

(:MODEL (INTERFACE TREE WELL))) 

(MISSING (INTERFACE TREE WELL) (VLAN E10)))). 


Tha conjunction of pradicatas collactivaly callad SEQ (on the arc from 
conjunction to sequential) plays a role in the following example. 
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Q6 ’ Reverse? tHe invocation order < TREE WELE > used, rather than the 


A6. TREE ends at its bottom, a required connection point, resultino 
that 1 ?^ 61 " interfacing for that ordering. If the TREE began at 

Referable l0n P ° int ’ reverse order "<>uld have been 


Here one of the SEQ predicates incorporates knowledge about the domain predicate 

CONNECTED, that interfacing can be simplified if two subprocedures are invoked in 

an order such that the endpoint of the first corresponds to a mutual connection 

point. An instance of this rule becomes a pragmatic assertion of the SEQ node in 
the parse. 


TREE^nd^n/ 0 / pre / er r ing the <™ EE VSLL} sequencing is that 
inet ends at a required connection point of WELL. 


(REASON (SEQ (E07 E08)) 

(AND 

(EQ (POSITION :TURTLE (AFTER TREE)) 
(BOTTOM (ENDPOINT TREE))) 

(EQ (POSITION :TURTLE (AFTER TREE)) 
(MIDDLE (UPPER (SIDE WELL)))))) 


A precise definition of a linguistic approach to protocol analysis has 

been provided end a concrete analysis of this kind supplied. We now turn our 

attention to the potential utility of the approach for constructing cognitive 
models of individuals. 
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——Toward a Cognitive Model of the Individual 


3.1. Tailoring the ATN to the Individual 

3.2. Individual Differences and Overlay Modeling 

3.3. Issues and Examples, and the Computer as Coach 


—1 * Tail oring the ATN to the Individual 

Advocates of computer-aided instruction point out that computers can be 
used to tailor instruction to the needs of the individual. Yet little is known 
about what it means to construct cognitive models of individual students, or 

about how to use them in providing sensitive and effective automatic tutoring. 
The SPADE theory suggests an approach. 


SPADE confronts the problem of individual differences by considering the 
possible ways in which the student's ATN can differ from that of an expert. One 
error would be to have a variant of the optimal pragmatic arc constraints. More 
serious would be to have missing or extra arcs. Even more serious would be to 
have missing or extra states. Differences which can be formalized as alterations 
to the topology of the ATN are manifested in the production of a different set of 
parse trees: PATN might be capable of some derivations not available to the 
student, or vice versa. Differences in arc conditions or arc actions are 
manifested by the selection of other than the optimal plan for a particular 
problem situation, although the same repertoire of plans may be available. 


These types of modifications, properly combined, can account for many 
commonly observed weaknesses in student problem solving. To demonstrate this 
point, we present six examples of student weaknesses and the fashion in which our 

modeling scheme is able to capture them. The examples are derived from informal 
data collected in our prior Logo tutoring experiences. 
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1 . 


%? C n£tt r r s A student with prior programming experience in 
the BASIC language never uses recursion. Problems for which 

forThich iteration d iTi °" ly With difficu1 ^; problems 

deep “ dra ’ ,1 "° arbUrarily 


A deviant version of the PATN subgraph for repetition planning is 

illustrated in figure 1:11. The correct suograph has an intermediate ROUND plan 

state; the deviant version, missing this state and its associated arcs, 

characterizes the BASIC syndrome. The student's repetition arc bypasses the 

ROUND state, short circuiting the ATN to pursue the iteration option with no 

possibility of recursion. In general, failure to employ a full repertoire of 

planning options can be modeled in this fashion: the short circuit is postulated 

to occur at the node immediately prior to the least common superset of the class 
of unused plans. 


2. Discontinuity: A student fails to build upon previous work 

Hew e Dict^re n9 to d h an H a9e °f relevant listing procedures. Each 
new picture to be drawn is treated as an isolated problem and 

recurring subproblems are repeatedly solved afresh. 


PATN can accomplish identifications using either primitives or previously 
solved problems. Discontinuity amounts to examining the primitive library only. 
This is modeled by the absence of the corresponding predicate on the arc from 
PLAN to IDENTIFY. A similar but more subtle case would be a student that 
occasionally uses previous solutions, but not as often as PATN predicts. This 
indicates that the identification network is probably intact, but parts of the 
reformulation subgraph are missing. Such a student fails to notice the relevance 
of previous problems because they are described in slightly different terms. 
Introspection suggests that this is a common source of difficulty. 
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Dotted lines are missing arcs. 
Dotted circles are missing states. 


FIGURE —I;11 DEVIANT ATN SUBGRAPH FOR REPETITION PLANS 
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3. ^gnosis Avoidance: A student performs well in planning and 

p D r °9 rams - However, when a bug occurs, the student 
if..!! ; * ather than systematically localizing the underlying 

of «. th ® arror ' Allowed by repair, the student immediately 

r.« 9 + to edit the Program. The changes are haphazard and 
counterproductive, creating more bugs than they eliminate. 


Relative to the DAPR debugging ATM, diagnosis avoidance is a weakness 
wherein the student has an extra arc not present in the expert. Whereas DAPR 
cannot proceed to the REPAIR state without first passing through DIAGNOSIS, the 
student is modeled as having an undesirable extra arc bypassing this state 
(figure 1:12). This allows diagnosis to be (incorrectly) treated as optional. 


4. 


Syntactically Unstructured Code: A student never uses 
subprocedures, instead relying entirely on in-line code. This 
results in long, unreadable programs which are difficult to 

^o?H 9 A ° ft r the student forgets which subgoals have been 
solved, or forgets how previously solved code segments work 
Few projects are successfully completed. 


PATN's use of subprocedures is governed by register setting actions 

associated with the sequential refinement loop. This is the culpable locale for 

♦ 

a type of non-modular design we call syntactically unstructured code. Instead of 
first setting the :PLAN register to a sequence of subprocedure calls, and then 
pushing for a solution to each in turn, the student apparently performs these 
actions in the reverse order: first pushing for a solution to each subprocedure, 
and then setting the :PLAN register to the concatenation of the popped results. 
Note that this deviant ordering of arc actions requires far more intermediate 
storage to keep track of recursive calls to the ATN: given a limited pushdown 
stack, it is not surprising that the student forgets things. 




tTaJl*) Cally yn5tru ctured Code: A student mechanically begins 
every Logo procedure with the PENUP command. Usually this works 

21 * "' ll - i" preparation for a petition tatnp. However, even 
when the position setup is unnecessary, the PENUP is still used 
resulting in either: (a) a rational form violation, in which 
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the PENUP is followed immediately by a PENDOWN; or (b) one or 
more missing model part violations, due to the invisibility of 
vectors intended to accomplish main steps. (Solomon [1976] uses 
the term clichk to describe this class of phenomena.) 

Treating some optional constituent as if it is required results in a 
second kind of non-modularity, semantically unstructured code . The particular 
cliche just described, mechanically including PENUP commands, is mediated by the 
linear decomposition arc. If this arc is modified to create position setup 

i 

subgoals without testing whether :M0DEL actually requires such a setup, the 
effect is to include a PENUP at the start of each turtle program. 


6. Pragmatically Unstructured Code: A student who normally does 
break large programs into subprocedures nevertheless encounters 
numerous bugs, many of which are difficult to localize. The 
subprocedures lack modularity, each being dependent on knowledge 
of the inner workings of others. For example, interfaces are 
included as part of main steps, so that the initial state of a 
given procedure is determined by the final state of whichever 
procedure happens to precede it in the planned order of 
invocation. 


While failure of a particular arc action to consider the problem at hand 
results in semantically unstructured code, faulty arc predicates in deciding 
among alternative arcs leads to a third form of non-modularity, pragmatically 
unstructured code. The unnecessary construction of non-linear subprocedures is 
attributable to either improper default ordering or malfunctioning predicates on 
the arcs leaving the conjunction node. For example, the INTERACTIONS predicate 
may not be imposing sufficiently strong conditions on accepting the model: this 

i 

* 

leads to the addition of constraints on the subprocedures when no real non¬ 
linearity is present. 

Thus perturbations on PATN provide a deep theory of student weaknesses, 
explaining unsuccessful behavior in terms of the syntactic, semantic, and 






Protocol Analysis 1.33 Miller & Goldstein 

pragmatic structure of the ATN. 

3.2. Individual Differences and Overlay Modeling 

We envision inducing a model of individual idiosyncrasies as 
perturbations of PATN by applying Goldstein's [1976] overlay modeling technique. 
This approach describes individuals with respect to an expert problem solving 
program by associating probabilities with each decision point in the expert, 
representing our state of knowledge about a given individual's preferences. The 
probabilities are a summary of the available evidence rather than an integral 
part of the model: at any given time, a process model is obtained from the 
overlay probabilities by including those possibilities that are above threshold 
and excluding those that are below. 13 Goldstein and Carr [1977] use this 

technique to infer process models of behavior in a logic and probability game 
called WUMPUS. 

This raises the question of whether all of the perturbations mentioned 
above, including the alterations in ATN topology, can in fact be represented by 
such an overlay, i.e., by a numerical plausibility table : it turns out that they 
can. A missing arc can be handled by assigning it an a priori transition 
plausibility of 0. A missing intermediate state can likewise be represented by 
the plausibility of the arcs leading to the unused states being 0, but the 
plausibility of the arc to the "short circuited" state being 1. Similarly, 

i 

default orderings can be reversed by reversing their relative plausibilities. 

This table driven organization allows distinguishing between personal and 
archetypal ATN's. Archetypal ATN's are analogous to Winston's [1970] concept 
models, and in fact our scheme for inducing personalized ATN's bears some 
resemblance to Winston's learning system, except that our networks happen to have 
procedural rather than structural meanings. Personalized ATN's are created from 
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the archetype by thresholding over a particular plausibility table. This 
simplifies the tuning and debugging of the system and eliminates the danger that 
states or arcs added to model non-expert behaviors might degrade the performance 
of the expert. The expert ATN is obtained by coupling the archetype with an 
expert plausibility table. The entries for undesirable options are zeroes. 

A first approximation to the plausibility table for an individual can be 
derived from the relative frequencies of arc transitions in previously analzyed 
protocols. However, this ignores the semantic and pragmatic context. It could 
be that infrequently used transitions were inappropriate for the tasks performed. 
Consequently, this is refined by comparing the tallies to a record of the 
expert's performance over the same set of tasks. (This technique, differential 
modeling, is suggested by Burton & Brown [1976].) Naturally there will be 
differences in individual protocols because of arbitrary choices, but in the long 
term consistent properties of the student's behavior should emerge. 

Just recording arc transitions is still too crude. One should account 
for differences in terms of the smallest chunks of malfunctioning knowledge which 
can be isolated. As a second order cognitive model, the units of analysis are 
taken to be the individual arc predicates and actions. The statistical evidence 
can be used to differentiate which arc operations are malfunctioning or missing. 

3.3. Issues and Examples, and the Computer as Coach 

Two crucial ingredients are lacking in current uses of computers in 
education: a cognitive theory describing the problem solving and learning 
processes, and a pedagogical theory prescribing techniques to facilitate and 
enhance these processes. As a result, many instructional applications of 
computers are ad hoc, if not detrimental. 

There are exceptions to these criticisms. The Logo project [Papert 1971] 
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offers educational applications of computer technology suggested by a 
computational approach to problem solving and learning. However, the 
justification for many Logo insights remains informal and intuitive. The current 
work is an effort to increase the theoretical precision and experimental rigor of 
Logo research. Other exceptions include the work of John Seely Brown's group 
[Brown et al. 1974,1975; Burton & Brown 1976] on intelligent instructional 
systems for electronics (SOPHIE) and elementary mathematics (WEST), and that of 
Stansfield, Carr and Goldstein [Stansfield et al. 1976; Goldstein & Carr 1977] on 
an advisor for WUMPUS. The WEST tutor suggests a paradigm, also used in WUMPUS, 

in which issues (abstracted differences between expert and novice behavior) are 

» 

illustrated by concrete examples of their application to active learning 
situations. 

Given the cognitive modeling tools developed in this chapter, an issues- 
and-examples Logo tutor can be contemplated. When PATN's expectations are 
violated because of a difference between the expert and student versions of the 
ATN, then that issue can be raised with the student. This would extend the 
issues—and-examples paradigm of WEST and the conputer-as-coach paradigm of 
WUMPUS, not only by addressing a more difficult task domain, but also by 
elaborating the notion of issues, from abstractions of empirically selected 
features, to specific programmatic weaknesses. 

The theory would also constrain the order in which issues should be 
presented to the student. The topology of the ATN should be nearly right before 
pragmatic arc constraints are discussed. Likewise, the general form of the 
pragmatics should be correct before domain*specific arc critics are taught. 
Although many subtleties arise which are not touched on here, the approach takes 
a step toward theoretical foundations for computer tutors which provide 
sensitive, flexible, individual instruction in problem solving skills. 
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4. Notes to Book I 

1. SPADE is an acronym for Structured Planning and debugging. See 
[Goldstein. & Miller 1976a,b; Miller & Goldstein 1976a,b,c]. 

2. More accurately, the session transcript is a partial protocol. 
Considerable leverage is obtained by assuming that the dialogue occurs within the 
confines of a small, well-defined response menu: natural language processing 
need not be attempted. We recognize that thorough protocol analysis includes 
parallel examination of the subject's utterances during the session, eye movement 
data, retrospective accounts, and so on. Although our sole objective here is 
analysis of the session transcript, we intend to corroborate our analyses using 
these other sorts of evidence. 

3. Miller & Goldstein [1976b] used a context free problem solving gr amm ar 
to extract the constituent structure of a student's Logo protocol. That paper 
did not develop the more thorough view of analysis we describe in Book I of the 
current report. 

4. PATN is designed in [Goldstein & Miller 1976b]. It has not yet been 
implemented. The use of present tense throughout this document in describing both 
PATN and PAZATN is for readability only. 

5. For efficiency, some states with similar topology are merged, and a 
few additional arcs are added to provide for such features as iterative control, 
when recursively invoking the complete system is unnecessary. 

6. The figure adopts a parenthesized notation (which is formally 
equivalent to that used in our earlier papers) to emphasize that predicate models 
are just LISP S-expressions which can be evaluated. 

At first these predicate models will be supplied by the experimenter. 
Eventually we plan to construct a module to induce the model from a hand-drawn 
tablet sketch. A significant undertaking itself, this would enhance the 
practicality of automatic protocol analysis in the graphics domain. 

7. Generation of pragmatic assertions representing instances of arc 
predicates is an elaboration of the basic PATN design, not presented in 
[Goldstein & Miller 1976b]. These assertions, being directly computable by 
examining the ATN's arcs and the semantic variables, are synthetically redundant, 
but become important when analytic complexities such as irrational bugs and 
personalized ATN's are considered. 

8. The example is a simplified hypothetical protocol not involving 
careless errors such as mistypings. In other respects, however, it is typical of 
student protocols for tasks similar to WISHINGWELL. 
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9. The root of the parse tree is shown at the left; the leaves are to 
the right. Some details are not shown: ellipses are indicated by three periods. 
For clarity, some semantic information is included parenthetically: SOLVE(WELL). 

Logo punctuation events are of minor importance in the underlying plan. 

Although used as clues during parsing, they are not included in the structural 

description. In the figure they are shown enclosed in exclamation marks: 
!E05 >END!. 

Since the order in which subgoals are solved need not mirror their 
execution order in the resulting plan, events need not occur in the parse in 

temporal order. In the figure the events are shown in temporal order, but lines 
are crossed. 

10. To illustrate the insights gained from the analysis, we use a 
scenario for a question-answering module which performs retrievals and simple 
inferences over a database consisting of the analyzed protocol. We are confident 
that the data structures generated by our style of analysis are sufficient to 
support this type of interaction. However, we have not yet designed the 
question-answering module per se; instead, we have concentrated on isolating the 
relevant knowledge base. For readability, the questions and answers are stated 
here in unrestricted English; for ease of implementation, the actual system will 
be restricted to a formal query language. 

11. It might seem that this definition of pragmatic annotation is 
inadequate for protocol analysis, since a student may select the right plan but 
for the wrong reason. The SPADE approach handles this circumstance by a separate 
mechanism, personalized ATN's, to be discussed shortly. For ease of 

presentation, the example uses the expert ATN as the basis for its REASON 
assertions. 

12. Although PATN's default solution to the wishingwell task did not 
involve debugging, PATN is capable of rational bugs such as this particular 
incomplete plan. When solving novel tasks, it is sometimes more efficient to 
plan only for the main steps, with the interfaces being solved by subsequent 
debugging. During planning, PATN notes those points where the plan is 

incomplete; when a bug is encountered, this advice guides PATN's debuqaina 
module, DAPR. 

Not all rational bugs are incomplete plans, and not all bugs are 
rational. Overlooking an interaction between subgoals is another type of 
rational bug. Mistypings and mispellings are typical irrational bugs; our 
approach to their analysis should be mentioned. The reason for such an event is 
assumed to be the same as the reason for the correct version of the event, but 
flagged by an additional assertion stating the nature of the mistake. 

13. Of course, one can also use probabilities to model actual non¬ 
determinism in the subject's behavior, but we do not consider that possibility 
here • 
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Book II: Automating the Protocol Parsing Process 
5. Introduction to Book II 

5.1. The CMU Series of Analyzers 

5.2. Overview of PAZATN 

Book I developed the SPADE notion of protocol analysis as parsing, but 
did not Indicate how parses are to be derived. Automating the analysis process 
is desirable, because manual analysis is informal, tedious, error prone, and not 
amenable to incorporation into computerized tutors. Hence, this second book 
presents the design for PAZATN, an automatic protocol analyzer based on the SPADE 
theory. As background for assessing the design of PAZATN, we first summarize the 
features and limitations of a series of automatic protocol analyzers developed at 
Carnegie-Mellon University. 

5.1. The CMU Series of Analyzers 

Much ground-breaking research in automatic protocol analysis has been 
performed at Carnegie-Mellon University. PAS-I [Waterman & Newell 1972], the 
of three CMU systems, analyzes think-aloud protocols for cryptarithmetic. 
PAS-II [Waterman & Newell 1973] is an interactive version which makes fewer task- 
specific assumptions. SAPA [Bhaskar & Simon 1976] addresses the additional 
complexities of semantically rich task domains. 

By focusing on the cryptarithmetic task, PAS-I obtains sufficient 
leverage to completely automate the analysis process. The input to PAS-I is a 
transcription of a tape recorded think-aloud protocol and its output is a problem 
behavior graph. PAS-I operates in four stages, the first two of which occur 
sequentially in time: linguistic analysis, semantic analysis, processing of 
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operator oroups, and problem behavior graph generation. PAS-1 does not attempt 
generality; for example, the linguistic analyzer employs a key word grammar 
oriented to cryptarlthmetlc. Similarly. Process-Column is a typical operator. 

PAS-11 reduces dependency on a single domain by requesting guidance, from 

a human encoder. Task-specific knowledge Is factered into a separate set of 

rules; the domain independent part of the system ameunts to a command language 

or subroutine library to assist a human protocol-analyst, (loving from automatic 

to interactive analysis may seem counter to progress. However, this 

methodological contribution allows flexibility to incorporate the experimenter's 

insight, while still Imposing discipline on the encedlng process. We intend to 

construct an interactive analyzer as an intermediate milestone in implementing 
PAZATN. 

SAPA, in cooperation with a human encoder, analyzes protocols in chemical 

engineering thermodynamics. By considering a domain rich in background 

knowledge, rather than puzzle problems such as cryptarlthmetlc, SAPA addresses a 

complex new facet of problem solving. However, SAPA is highly domain specific. 

For example, SAPA begins the analysis by asking for the form of the energy 

equation used by the subject. Thermodynamics problem solving is viewed as a 

variant of means-ends analysis in which the energy equation plays a predominate 
role. 


When implemented, PAZATN will extend the automatic protocol analysis 
techniques developed at CMU by complementing their features and limitations. On 
one hand, a PAZATN shortcoming - its restriction to a small menu of responses - 
is addressed by the considerable effort CHU researchers have invested in natural 
language front-ends for protocol analysis. On the other hand. CHU has devoted 
less attention to the investigation of planning concepts, a limitation addressed 
Ky the SPADE theory. For example, the CHU theory does not provide a deep account 
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Of the origins of planning errors in the PATN sense. Likewise, a practical 
limitation of the CMU analysers has been task specificity. In designing PAZATN 
KO have tried to minimise task specificity throogh modular design; this is mad. 
possible, in part, by the highly structured underlying SPADE theory. However, 
testing the generality of PA2ATN by applying it to several domains remains a 
research goal. Finally, the elementary programming world to which PAZATN is 

applied in this paper resembles thermodynamics in that background knowledge of 
the domain plays a significant role in solving problems. 

5.2. Overview of PAZATN 

PAZATN is a scheme for matching a protocol to a PATN plan derivation; it 
can only understand protocols which PATN can generate. 2 Therefore the analysis 
could be performed, in principle, by trying all possible PATN solutions, 
selecting the first which matches the data. Since exhaustive enumeration is 
impractical, a primary consideration is efficient search in PATN's plan space. 
Bottom-up protocol evidence is used for this purpose (figure 1:1). 

PAZATN consists of PATN supplemented by several additional modules and 
data structures (figure II;1). This design incorporates three key ideas: 

*• “n di , atinc?*ro C i*.', r * f r “' t “ re ^ Kaplan 1973] in 

ef!w fll n 1 roles, both involving the need to economically 
store alternative combinations of substructures; cononica lly 

2 ' J?o e cessino°Lo,i?n r “ ri ',° / ll '"«‘»-*P«ci/ic specialists for 
processing events in various syntactic categories; 

3 ‘ th ! ^ S , e of be5t * irst coroutine search driven bv a 

scheduler — with modules communicating by means of the charts 
- to ensure early application of strong sources of constant 

1. Two charts, One of PAZATN's charts, the planchart, keeps track of 
subgoals proposed by PATN. PAZATN's second chart, the dat.chart, records the 
alternative ways of associating protocol events with planchart leaves. 





MODEL 



PROTOCOL 
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2. Library of event specialists, The syntactic classification of 
possible events for a given domain results in a highly modular design. (ADDCODE, 

UNCODE and END are typical Logo event types.) For each event type PAZATN is 

supplied with an ESP, i.e., a specialist for associating events of that type with 

planchart leaves. Adapting PAZATN to other problem domains is possible by 
replacing this library. 

3. Best first coroutine search, PAZATN receives the model and protocol 
as input. The model is a formal statement of the problem as shown in figure 1:7; 
the protocol is a list of events as shown at the top of page 1.19. PAZATN's 
output is a SPADE interpretation of the protocol as described in Book I: a parse 
tree augmented by semantic and pragmatic annotation. At any given time during 
analysis, several partial interpretations will be active. The outer loop is a 
scheduler which allows each active partial interpretation to examine one event 
per cycle. For a given interpretation, events are processed in a single left-to- 
right pass. At the end of a cycle the active set is re-chosen. This repeats 
until at least one interpretation has processed the final event. 

Analysis of a protocol proceeds as follows. First PAZATN requests PATN 

to generate its most plausible plan on the basis of the model alone. This plan 

is inserted into the planchart. Next, protocol events are examined one by one, 

matching them with subgoals in the PATN plan. Each match is recorded in the 
datachart. 

If an event is encountered for which no plausible match can be found, 
PATN is asked to generate its next most plausible plan, now potentially 
considering the nature of the mismatch as well as the model. The planchart is 

extended by inserting PATN's next plan. Those subgoals which are common to both 
plans share the same structure in the chart. 
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If an ev ant Is encountered for which more than one plauaible match can be 

found, the datachart records each such pairing in similar fashion. Each of these 

is then allowed to continue examining protocol events according to the best first 
scheduling algorithm. 
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_Simulating Automatic Parsing of the Example Protocol 


6.1. Preliminary Generation of Expectations 

6.2. Modifications Based on Bottom-up Evidence 

6.3. Some Informal Observations 


This chapter is a hand-simulation of PAZATN on the WISHINGWELL protocol 
introduced in Book I. Various components of PAZATN are introduced as they are 
needed. Subsequent chapters provide details regarding these components. 

6.1. Preliminary Generation of Expectations 

The protocol parsing process is initiated by executing (PAZATN 
VISHINGWELL WWJ, where WISHINGWELL is the model and WW the protocol. The initial 
answer library is assumed to contain procedures for TRIANGLE and TREE. 

Before PAZATN examines the protocol, PATN examines the model. Since 
WISHINGWELL is not in the answer library, PATN determines that an identification 

plan is not viable. Both decomposition and reformulation are possible, since 
they are applicable to any model. 

PATN can determine, using lookahead, that reformulation results in an 
identification involving TREE; for this particular protocol, this quickly leads 
to a successful parse. However, reformulations rapidly expand the search space, 
so PAZATN adopts a conservative approach to reformulation: decompositions which 
lead to a straightforward solution are preferred unless protocol evidence 
indicating reformulation is discovered. Consequently decomposition is predicted, 
with three main steps: ROOF, POLE, and WELL. But since the decision is 
uncertain, a demon procedure 3 is created to handle the possibility that 
decomposition fails to parse the protocol. 

The model is examined for interactions. None are detected, so a linear 
decomposition into subgoals is expected. However, since required connection 
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points occur at the midpoints of sldas of WEIL and ROOF, a non-linear subgoal 
decomposition might be used for efficiency, to avoid retracing. As a result, two 
demon procedures are created to check for WELL or ROOF sides being accomplished 

steps. Such a plan is less likely for ROOF which can be identified with 
the existing TRIANGLE. 


The transitive ABOVE predicates suggest a sequential plan utilizing 
either the order. (ROOF POLE WELL), or the order, (WELL POLE ROOF). There is no 
basis for selection. Hence. PATH follows a principle of least commitment, 
predicting the disjunction of the two invocation orders. 

This application of the principle of least commitment is accomplished 
using a chort" of alternative plan derivations called the pioncAart. The 
planchart is similar to an MO/OR ,ool tree but involves a variety of node types 
and shares substructures economically. Figure 11:2 illustrates how the two 
equally likely sequences are represented in the planchart. As PATH generates 
predictions, the required bookkeeping is performed by expanding this planchart. 

Since the main steps for the two sequences are identical, they provide no 
evidence regarding ordering. The interfaces provide the critical evidence, so 
PATH solves the interfaces for one order, (WELL POLE ROOF). Because the choice 
is arbitrary, another demon is created to expand the (ROOF POLE WELL) order in 

case the interfaces fail to match. Except that TRIANGLE is already in the answer 
library, PATN has predicted the protocol of figure 1:8. 


Besides predicting PATH'S default solution, three arbitrary choices have 
been flagged as likely failure points. If the specific discrepancy pattern 
monitored by one of the three corresponding demons is detected, that choice will 


be reconsidered. If non-specific mismatches are encountered, backup to other 
decisions will occur in the usual way. Note that most choices are not arbitrary 
and have not been flagged. (This helps to avoid the usual inefficiency 
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•SOLVE (WELL) 




SEi 


SOLVE (INTERFACE 1) 


OLVE (ROOF) 


OLVE (INTERFACE 2) 





'OLVE (POLE) 




SOLVE (WW)-. CONJ . . .—fSPLIT 


SEQ 




OLVE (POLE) 


OLVE (INTERFACE 3) 


OLVE (ROOF) 


OLVE (INTERFACE 4) 


OLVE (WELL) 


FIGURE 11:2 SIMPLE PLANCHART FOR ALTERNATIVE ORDERS 
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associated with pure backtracking control structure: "failing in all possible 
ways.") 

At this point, control passes to the bottom-up analytic routines. 

6.2. Modifi cations Based on Bottom-up Evidence 

PAZATN now attempts to interpret the first protocol event in a manner 
consistent with PATN's default solution. 

E01 ?TO WELL 

E01 is classified as a TO event -- Logo punctuation beginning a procedure 

definition. The event specialist for TO events is called upon to assign the 
event to some expectation. 

PAZATN does not use mnemonic clues, and no significance is attached to 
the student's particular choice of the name WELL. 5 The TO specialist examines the 
planchart (figure 11:2) for candidate subprocedures. There are expectations for 
the top level (WW), WELL, POLE and the two interfaces. The default solution 
order is top-down, so E01 is assumed to start WW. However, solution order is so 
variable that other interpretations are plausible. Consequently the 
interpretation splits into separate analyses for each. 

Whereas the planchart is used to keep track of alternative expectations, 
a second chart, the datachart, is used to keep track of alternative associations 
between protocol events and expectations. PATN expands the planchart; PAZATN's 
event interpreter expands the datachart. At any given time, some of the partial 
interpretations in the datachart are considered to be active; the rest are hung. 

For expository purposes, we will assume that only one partial 
interpretation is active at a time. Rather than pursuing several alternatives in 
parallel, we will merely record them in case the need to back up arises. Hence 
after the split is performed, E01 is assigned to be the TO for the top level 
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procedure, WW. The parent node of this interpretation, a generator for 
alternative interpretations of E01, is hung. 

With E01 assumed to start WW, E02 is now processed. 

E02 >10 REPEAT 4 [20 30] 

This does not match the expectation for a definition of the top level WW 

procedure. Therefore, backup occurs to the most recent split (at E01). The only 

alternative that can account for E02, that E01 is the start of WELL, is activated 
(figure 11:3). 

The protocol matches this new interpretation through E05. 

E03 >20 FORWARD 100 

E04 >30 RIGHT 90 

E05 >END 

Ambiguity arises at E06. 

* 

E06 >TO WW 

Since ROOF can be identified with TRIANGLE and WELL has already been found, this 

must be WW, POLE or an interface. The POLE and interfaces are apt to be solved 

by in-line code; furthermore, top-down order is the default preference. Hence, 

although E06 causes a split, WW is clearly chosen as the active interpretation. 
Next, E07 is examined. 

E07 >10 TREE 

Rather than matching WW's expectations for a setup or a call to WELL, E07 
matches the discrepancy pattern for two active demons. One demon represents the 
possibility that the (ROOF POLE WELL} order was used; this would require TREE to 
be the setup for ROOF. The other demon represents a potential reformulation 
involving TREE. This second demon is highly specific for this evidence and is 

therefore triggered. Control returns to PATN with a request for a reformulated 
model in which TREE is a subgoal. 

PATN regroups ROOF and POLE into TREE, and then expands for a solution to 
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E01=WELL • 
DEF'N 


E02=REPEAT4 

ACTIVE 


FIGURE 11:3 DATACHART AT E02 OF WW 
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the revised model. When the sequential refinement loop is reached, the 

{TREE WELL) invocation order is chosen immediately on the basis of known protocol 

data. This need not have been PATH'S choice from a problem solving point of 

view, this decision is forced by the bottom-up evidence. Figure 11:4 shows the 
modified planchart. 


After PATN has processed the reformulation request, E07 can be 

accommodated, as shown in the datachart of figure 11:5. EOS is now examined. An 
interface is expected. 

E08 >20 WELL 

WELL is known to be a previously solved mainstep, violating that expectation. 

This is the standard pattern for an Incomplete plan: an interface is expected 

but instead the next mainstep is found. A demon for incomplete plans is always 

active and is triggered by this situation. It passes control to DAPR which 
generates debugging expectations. 


Each remaining event matches a DAPR expectation. 

E09 >END 

E10 >?WW 

Ell >?EDIT WW 

E12 >13 RIGHT 90 

E13 >15 FORWARD 100 

E14 >17 RIGHT 180 

E15 >END 

Hence the parse succeeds. Figure 11:6 shows the final planchart and datachart, 
with marked nodes indicating the parse tree which is returned. 


6.3. Some Informal Observations 

We have hand-simulated PAZATN on about a half-dozen hypothetical 
protocols. This informal exercising of the design has led us to a number of 
tentative observations regarding PAZATN's capabilities. One question which 
arises is PAZATN's flexibility to handle alternative solutions. We are confident 
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FIGURE 11:4 PLANCHART AT E07, AFTER REFORMULATION 
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FIGURE 11:5 DATACHART AT E07, AFTER REFORMULATION 
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that the following types of variation can he handled: 

1. subprocedures versus in-line code; 

2. incomplete plans where interfaces are solved by debugging; 

3- <b c u° t rr k r„™ p by n PA?5;r lnt,ri " :tions are ° variooke<i * «= 

4. permutations of invocation or solution order; 

5 ’ c\nve?sion); ref0rraUlati0nS (regrou P in 9’ generic-explicit 

6 ' efficiency n ° nlinear ^compositions (accidental or for 

7. non-standard default parameters (FORWARD 75 as the basic unit); 

8. simple forms of equivalence (BACK 100 versus FORWARD -100); 

9. common errors such as mistyping, or omission of a line number. 

On the other hand, the following types of variation pose problems for PAZATN: 

1 . interleaving of lines from different procedures if errors also 
occur in that a procedure is accidentally edited; 

2. unrecognizable reformulations due to gaps in PATN's knowledge; 

3 ’ o^UoCC; 17 ° bSCUre C ° d0 ’ ° r C ° de involving “any needless 

4. equivalence transformations resting on subtle domain theorems; 

5. fully general recursion including heterarchical procedure calls. 

Another observation concerns PAZATN's efficiency. For the simple 

protocols we have considered, after only a few false starts, PAZATN latches onto 

a correct set of expectations regarding the student's overall plan. After that 

point (which we would place at E08 for this protocol) interpretation of the 
remaining events proceeds without incident. 
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7. Organi zation of the PAZATN Protocol Parser 

7.1. The Planchart 

7.2. Representing Interpretations 

7.3. The Datachart 

7.4. Incremental Planchart Expansion 

7.5. Markers and Marker Propagation 

7.6. Preprocessing 

7.7. The Event Classifier 

7.8. The Event Interpreter 

7.9. The Event Specialists 

7.10. The Scheduler 

In generating potential protocol interpretations, PATN is guided not only 

by synthetic evidence derived from examining the model, but also by analytic 

evidence derived from previously examined protocol events. If previous events 

have established that the student is pursuing a particular subgoal, then PATN 

will propose candidate solutions for that subgoal, even if it is not one which 

arises in PATN's preferred plan. Likewise if previous events have established 

that the student is pursuing a particular invocation order, then PATN will use 

that order in creating interfaces, even if another sequence leads to simpler 

interfaces. This sensitivity to the student's plan is accomplished by adding 

additional predicates to PATN's arcs which access assertions in the current 
partial interpretation. 

This chapter presents the major PAZATN modules needed to use PATN in this 
analytic role. Chapter eight refines the discussion presented here. 

7.1. The Planchart 

PATN is an intentional representation of the plan space; there are a 
number of reasons for needing an entensionol representation of the ATN process. 
Consequently a complete trace of PATN's operation, the plonchort, is maintained. 
One reason for creating this data structure is to avoid repetitive calculations, 
but additional uses for the planchart will appear in the course of the 
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discussion. (Figure 11:4 shows an example planchart from the analysis of WW.) 

The planchart includes not only plans, but nodes of other types such as 
debugging episodes. As its name suggests, the planchart is a chart [Kay 1973; 
Kaplan 1973], a network which compactly represents alternative combinations of 
subexpressions. This economically represents PATN's partial solutions and their 
hierarchical annotation. Rather than generating the entire solution space at 
once -- which would be impractical even if it happened to be finite -- PATN 
expands this planchart incrementally as additional possibilities are needed by 
the analyzer. 

Looking upward from a given leaf, the planchart resembles an AND/OR goal 
tree. However, there are a greater variety of node types, rather than just AND 
and OR. This allows the planchart to represent such concepts as whether 
conjunctive subgoals need to be accomplished in a specified order, or whether any 
order will do, allowing a greater variety of potential interpretations to be 
expressed parsimoniously. 

The analysis process is closely tied to modifications of this data 
structure. In particular, the structural description assigned to a protocol 
corresponds to a pathway through the planchart starting from the root -- the top 
level SOLVE node -- to the individual protocol events corresponding to a subset 
of the leaves. The semantic variables and pragmatic assertions are generated by 
^TN along with the parse, and are attached to the corresponding planchart 
nodes. Consequently, the structure building actions of the protocol parser are 
performed entirely by PATN. 
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An interpretation of an event is represented as an assignment of that 
event to a leaf of the planchart (figure 11:7). Similarly, an interpretation of 
the protocol is a complete association list of such event assignments. A partial 

interpretation is an association list containing assignments for a subset of the 
events in the complete protocol. 

Because of the chart representation of plans, individual events can be 

assigned to a single leaf but remain ambiguous as to which plan they belong to. 

The assignment captures exactly what can be concluded from the event: no more 

and no less. All possible interpretations consistent with the data are carried 
along. 

In order to be assigned to a given leaf of the planchart, it is not 

necessary for the protocol event to match identically. Data events are converted 

to canonical form before assignment, so that equivalent forms (e.g., LEFT 90 and 

RIGHT 270) are not distinguished. Non-equivalent assignments are also possible, 

representing the analyzer's judgment that the protocol event was intended to 

match the planchart leaf but contains either errors, such as mispellings or 

mistypings, or different default parameters where a range of values is 
acceptable. 

7.3. The Datachart 

A partial interpretation splits when it proposes more than a single 
planchart assignment for an event. Some method for keeping track of the 
analyzer's alternative partial Interpretations is needed. It should take 
advantage of the fact that, following a split, the event interpretations prior to 
that split remain the same: the common ancestry should be preserved. Ideally 
interpretations which agree on events both he/ore and o/ter a split should share 
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E03 has been assigned to the planchart generic side for WELL. 


FIGURE 11:7 INTERPRETING AN EVENT 
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the same representation for them; this is called a join. 

The datachart serves these functions. Like the planchart, the datachart 
is a chart, so that it can economically store common substructure. Suppose that 
two interpretations have identical assignments for the first M events, and then 
split. The split corresponds to a single node having two descendants. 
Assertions corresponding to the shared part of the interpretation are 


automatically inherited from the parent node (figure 11:8). 

Whenever a low plausibility event assignment occurs the following actions 
are performed: 


lm MsianleVrii* f l" 8 CUrTent node ’ indica *ing which event 

no^fhiH^ 1 f 1 b , 0Ut t w be made * This ensures that the same 
possibilities will not be repeatedly pursued. 

2 ' Ve" 1 „*„S r0M S: d / " hiCh “ 1U ‘ nher lt Prior assignments from 
nnroJJr? 4 5 ? This ensures that changes which reflect the 

the parent node 3 "°* aff8ct the statB ‘»r«™.ti«n of 


The 


3 ‘ I!l®ma n i Certaln assi 9 nment is performed at the new node. 

hTkT?" 5 associated w ith event interpretation 
(described below) are carried out. 

4 ‘ ^Mc neW n ° de It PlaC6d on 3 list of NEW P artial interpretations. 

iiTRSsr 11 be scheduied for at ieast “• cycie * 

5 * It® ?! r l nt node is re ' exa “ined to determine if additional nodes 

If la SP H r ° Uted re P resen ting alternative event assignments? 
If so, the above sequence of operations is carried out for each 

When no further alternatives seem worth considering at the 
interpretation's.** 16 Par6,lt n ° d6 iS PlaC8d ” a llst ° f HUNG 


This technique has the feature that it is not necessary to explicitly 
list all of the possible alternative interpretations for a given event. After 
sprouting, the parent node no longer represents a single partial interpretation, 
but an indefinite number of implicit alternatives to its current offspring. Even 
after it is HUNG, the parent node contains the necessary state information to 
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possible explanations of E07 can also be seen. 


FIGURE 11:8 INHERITANCE OF DATA CHART ASSERTIONS 
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generate additional possibilities if these are ever needed. 

7.4. Incremental Planchart Expansion 

Consider the situation in which an active partial interpretation cannot 

find an acceptable planchart assignment for its next event. Two conclusions are 

possible: either (a) the current partial interpretation is a dead end, and 

should be moved to the HUNG list; or (b) the current partial interpretation is 

viable, but the planchart has not been expanded sufficiently to account for the 
current data. 

This decision is crucial. If PAZATN is too miserly in allowing planchart 
growth, an event could be mis-interpreted as a deviant version of an existing 
leaf, when only slight growth would have allowed it to match a new leaf exactly. 

But if PAZATN is too eager to expand the planchart, the number of irrelevant 
solutions proposed could be enormous. 

This decision is also very difficult, being complicated by the 

circumstance that data events need not identically match planchart leaves: they 

can differ because of postulated bugs or variant but acceptable parameter values 
(such as scale factors). 

Four techniques are germane to this decision and its complications. 


1. Protocol events are converted to a canonical form. This allows 
versus Vao^ 5 ! 00*°^ 6 ^° rmS 07 e< l u l- v alence such as FORWARD -100 


2. Standard spelling correction procedures 7 are applied to 
unrecognized protocol events, using the fringe of the planchart 

mispellings nary ’ alloWS for handlin 9 simple mistypings and 

3. A hash coding scheme uses the critical terms of an event fe a 

varianfl^nf ^ ^ ^ 100) * S keys ’ 8 71,18 allows acceptable 
variants of events (e.g.,, those differing only by a scale 

factor) to be located. 

* * 

4. The neighbors of a planchart leaf provide expectations which 
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Th^nex" section" ° f 8Vent assl 9""»"‘t to that leaf. 

expectations -““tribes • scheme for generating these 

7 .5. Markers and Marker Proparp n™ 

A marker propagation technique helps in deciding whether to expand the 
planchart by providing a precise representation for expectations. Markers also 
determine the final protocol parse by selecting a pathway through tho planchart. 
Assigning a protocol event to a planchart leaf marks that leaf. Three types of 
markers are used: (1, a standard marker for events that match identically or 
differ only in a flexible parameter value; a distinguished marker for top 
down DEFINED plans prior to encountering the body of the subprocedure; and (3) a 
distinguished marker for deviant events involving mistypings or similar errors. 

A constituent is expected to the extent to which finding it results in 

propagations, where propagation through the planchart is characterised by rules 
such as: 

M pR - DI sj if the parent of a marked node is disjunctive <i e a 

split), the parent is marked; J a 

MPR CONJ. If the parent of a marked node is conjunctive (e a crrn an it 
every sibling of the marked node is JarVet tie” parent is* 

The rules shown here are incomplete. Top down DEFINED plans, for 
example, receive special treatment to ensure that after completing a 
superprocedure the expectations for its subprocedures remain in effect. 

As an example of the use of these rules, consider a bottom-up DEFINED 
Plan, where a subprocedure is first defined and then called by a superprocedure. 
After the subprocedure definition has been encountered, its use by some 
superprocedure is expected. The planchart would contain a marked SOLVE node for 
the subprocedure and an unmarked USE node for its use in the other procedure. 
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both dominated by an unordered conjunctive "&• node (figure 11:9). The USE is 

expected because marking its node would result in a propagation at least as far 
as the SOLVE dominating the DEFINED node. 

Suppose that an expectation (such as the bottom-up DEFINED plan example) 

fails to be satisfied after many events. One possibility is that the partial 

interpretation which expects it is completely wrong, and should be abandoned. A 

second possibility is that the partial interpretation is basically correct, but 

the student has accomplished the expected effect in an alternative way (e.g.. 

Incorporated the subprocedure's definition in-line instead of calling it as 

expected). This second case turns off the expectation, since it becomes 
dominated by a marked node (figure II: 10). 


A third possibility is that the student accidentally left out the 

relevant line of code. This is detected when protocol events indicate that the 

episode is finished. In the Logo world this corresponds to encountering the END 

statement for the superprocedure. END statements force propagations even when 

some expectations are not satisfied; but the plan is flagged as incomplete, 

debugging expectations are generated, and the plausibility is lowered. If the 

debugging predictions are then confirmed, the plausibility is restored and the 
expectation considered satisfied. 

Markers, as a representation for expectations, provide evidence regarding 
the plausibility of interpretations, which is especially useful when planchart 
expansion is under consideration. Typical plausibility guidelines include: 


PLG-1. Event assignments that result 
are more plausible than those 
propagations or none at all. 


in longer chains of propagations 
that result in shorter chains of 


PLG-2 


Interpretations that leave few 
plausible than those that leave 


expectations unsatisfied are more 
many expectations unsatisfied. 
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the propagations shown as?'s. 

--■ GURE 11:9 -DEFINED PLANS: AN EXAMPLE OF PROPAGATION AS EXPECTATION 
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Planchart 


Protocol 
TO WW 


X 

r-SOLVE (WELL) 


X 

SOLVE(WW)— 


X 

• • »SEQ 




V 


-SOLVE (*WELL*) 



—USE(WELL) 


I 

i 





in-line code for well 



END 


This use of WELL is no longer expected, 
since it is now dominated by a marked 

node. 

FIGURE 11:10 EXPECTATIONS CANCELLED, DOMINATED BY MARKED NODE 
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PAZATN includes a preprocessor which performs four functions. 

1. Low level syntactic anomalies such as typooranhicai or,™ 

onl^th 611 USin9 i he RUB0UT and BREAK keys are filtered out- 
y e corrected versions of such events are examined. 

2 ' l slV^\mZTn i0n , clues 1 are "°^ d - example, with raster 
?c r. J URTLES CLieberman 1976] global connectivity of vectors 
is readily detectable and suggests a segment boundary. 

3 * Ii^ 9 n data at L e , collected - This information may be of value in 

XVibilitv'., 0 ™'!'? 1 Cl . ,1 “’ and ln tem,*instances the 
betweeS t^e-lns " lnterPretatl ° n depends “■»" «>= lapsed time 

4 ‘ rirnmnf,? ry * 3 ? tac J ic class of e«h event is recorded to avoid 
recomputing it under each interpretation. Classification is 

performed by a separate module which can be re-intoked if the 
primary class is later called into question ® 

7.7. The Event Classifier 

The event classifier, one of the few PAZATN nodules which nust be 

redefined for each domain, contains the syntactic knowledge necessary to 

distinguish various donaln-specific event types. For the programing world, the 

event types include RUN events. EDIT events, and so on. In assigning an 

interpretation to an event, a variety of semantic and pragmatic evidence Is 

ultimately considered by PAZATN, but the event classifier Is restricted to 
syntactic evidence. 

The event classifier can be invoked In three modes. Normally It Is 
Invoked by the preprocessor, with its Input an event and its output the event's 
primary syntactic class; for most events, this Is sufficient. In the second 
mode it is invoked by partial interpretations which question the primary 
syntactic class, with a specific alternative class being considered. Here its 
input is an event and a class name; its output is a numerical score summarising 
the syntactic evidence supporting the alternative class. l„ the third mode the 
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classifier is invoked by partial interpretations which question the primary 
syntactic class but with no specific alternative class under consideration. An 
exhaustive rank ordered list of categories and scores is returned. 

Event classification will be performed using straightforward pattern 
matching. No further details are given here. 

7.8. The Event Interpreter 

The event interpreter is responsible for category independent operations 
of event interpretation. This includes the node sprouting sequence described in 
the datachart section, the processing required for marker propagation, and the 
plausibility computations. The rationale for grouping these activities is 
modularity: they are required for every category of event interpretation. 

The event interpreter is PAZATN's inner loop. It is invoked by the 
scheduler with two arguments: a partial interpretation, and a protocol event. 
It attempts, in cooperation with one or more event specialists, to account for 
the protocol event in the context of the partial interpretation. This can result 
in the creation of additional (descendant) partial interpretations. Control 
returns to the scheduler when event interpretation is complete. 

% 

7.9. The Event Specialists 

A collection of domain specific event specialists (ESP's) are responsible 

for category dependent operations of event interpretation. Each specialist 

contains the requisite knowledge for analyzing events of a particular syntactic 

type. The event interpreter invokes an ESP, in the context of a partial 

interpretation, with an event and an implicit assumption regarding its syntactic 

category. The specialist is free to assign any interpretation to the event which 
is consistent with the category. 

If the event specialist does not return with a sufficiently plausible 
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event assignment, the event interpreter then considers the possibility that the 
syntactic category postulated for the event is incorrect. Whenever an event is 
interpreted as being in error, expectations for diagnosis and repair are 
generated by DAPR at the request of the event interpreter. 

ESP's use a decision tree organization to factor the analysis into 
several cases. Each case represents an assumption about intent; if the 
assumption is uncertain, the state of the interpretation is preserved by 

sprouting a new datachart node. This is exemplified by the Logo ADDCODE ESP, 

whose flowchart appears in figure II:11. 10 

» 

The ADDCODE classification assumes that the current event is intended to 
add a new line of code to the procedure definition. Hence it must be determined 
whether Logo is actually in definition mode. If not, the following event will be 

an error message. If the ADDCODE assumption is correct despite the error, the 
current event will be repeated after a TO event. 

Lookahead is required to assign the current event to be an erroneous 
version of a later event. However, in a real time tutoring application, the 

i 

later event might not have occurred yet; moreover, processing more than one 
event would exceed the scheduler's resource allocation. This dilemma is resolved 
by creating a demon to represent the current event assignment. The demon will 
fire when the future event is assigned, assigning the now current event to be a 
deviant version of the later event. 

In the case where Logo is in definition mode, ADDCODE branches to one of 
the following subcases: (a) the added code is a turtle primitive; (b) the added 
code is a Logo control statement (such as a recursion or iteration line); (c) 
the added code is a call to a user procedure other than a recursion line. 
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7.10. The Scheduler 

The scheduler's task is to cause partial interpretations which have a 
reasonable likelihood of succeeding to make progress, and prevent those that are 
likely to fail from consuming valuable resources. Operationally this means 

driving PAZATN through a best first coroutine search of the space of partial 
interpretations. 

The search is accomplished by maintaining three lists of partial 
interpretations: NEW, ACTIVE, and HUNG. The scheduler cycles through the ACTIVE 
list, allowing each item to process one protocol event. Then the plausibility of 
each modified interpretation is recomputed, and the ACTIVE and HUNG lists are re¬ 
chosen. NEW interpretations, which result from the splitting of ACTIVE 
interpretations on the previous cycle, are then moved to the ACTIVE list, 
guarantying them at least one quantum of processing. The plausibility of a 
partial interpretation increases with each additional event accounted for. (This 
acts to decrease the relative plausibility of older HUNG interpretations.) 

This process continues until at least one ACTIVE interpretation has 
processed the last input event without unsatisfied expectations. If the first 
successful interpretation is not sufficiently better than every other candidate, 
some of the better alternatives are pursued until they become implausible or 
determine that the protocol may successfully be interpreted in more than one way. 
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8. Refining the Protocol Parser 

8.1. Lookahead 

8.2. Least Commitment 

8.3. Differential Diagnosis 

Our basic protocol parsing scheme is to generate expectations with PATN 

and then try to match these expectations to a protocol. This process is refined 

by several techniques which have enhanced the effectiveness of problem solving 

and language processing programs: lookahead (e.g., [Aho & Ullman 1972]), least 

commitment (e.g., [Sacerdoti 1975]) and differential diagnosis (e.g., 
[Rubin 1975]). 

8«1» Lookahead 

Lookahead and least commitment are related search strategies designed to 

avoid premature decisions based on inadequate evidence, which can result in 

needless backup. Lookahead consists of briefly examining subsequent input events 
before interpreting the current event. 

PAZATN can accomplish a limited form of lookahead by using demon 

procedures to represent event assignments. When the current event assignment 

depends upon a future event assignment, a demon is created which will complete 

the. current assignment when the missing evidence from the future assignment is 
available. 

8.2. Least Commitment 

Variability in solution order exemplifies the need for avoiding premature 
commitments. PATN always defines the top level plan before expanding 
subproblems, representing strict top-down problem solving (figure 11:12), but 
human programming is rarely this uniform. When the need for a particular subgoal 
has been established, it may be expanded immediately, prior to completing the top 
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SOLVE(WW) 


SEQ 

SOLVE(*ROOF*) 
DEFINED 
USE 
SOLVE(ROOF) 



SOLVE(*POLE*) 
DEFINED 

USE- 

SOLVE (POLE) 


SOLVE(*WELL*) 
DEFINED 
USE- 


SOLVE(WELL) 




V 


?TO 

WW 

'>10 

ROOF 

>20 

POLE 

>30 

WELL 

>END 

?TO 

ROOF 


; >END 


/ 


! ?TO POLE 


I >END 


f?TO WELL 


I 




>END 


FIGURE 11:12 A TOP-DOWN EXPANSION FOR WISHINGWELL 
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level plan, representing bottom-up problem solving (figure 11:13). 

Least commitment helps to minimize misleading mismatches between 

planchart and protocol resulting from different solution orders. This is 

accomplished by using unordered conjunctive nodes in the planchart. Thus 

when DEFINED plans are expanded to USE & SOLVE, the SOLVE may occur prior to the 
USE with no loss in plausibility. 

The least commitment policy is applied to variability in invocation order 

as well. When, as was the case with WW, more than one invocation order is 

acceptable, the planchart is split. This parallels the use of procedural nets 

[Sacerdoti 1975] to avoid overspecifying ordering constraints (figure 11:14). 

The chart data structure allows the ambiguity to be represented without 

significant additional cost: if the mainsteps are identical for both orders, 
then two copies will not be stored* 

Despite its virtues, though, least commitment could be overdone, 
resulting in so large a disjunction of expectations that no guidance would be 
obtained. PAZATN strikes a balance between overcommitting itself and refusing to 
take decisive action: it avoids arbitrary choices in the course of a given 
decomposition strategy, but adheres to a given formulation of the model unless 
required to change it by specific bottom-up evidence. 

8*3* Differential Diagnosis 

The use of demon procedures to implement lookahead was discussed earlier. 
Another use of demons is to perform differential diagnosis, using highly specific 
clues to distinguish between similar competing interpretations. The primary 
application of differential diagnosis demons is to the choice between assigning 
an event to one of an existing disjunction of expectations, and reformulating the 
problem description in response to bottom-up evidence. 
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SOLVE(WW) 


SEQ 

SOLVE(*ROOF*) 
DEFINED 

SOLVE(ROOF); 



USE- 

SOLVE(*POLE*) 
DEFINED 

SOLVE(POLE) 

USE-- 

SOLVE(*WELL*) 
DEFINED 

SOLVE(WELL)] 

i 

i 

• i 

i. 

J 

USE-- 



?TO ROOF 


>END 

?TO POLE 


>END 

?TO WELL 


>END 


?TO WELL 
>10 ROOF 
^>2 0 POLE 
->3 0 WELL 
>END 


FIGURE 11:13 A BOTTOM-UP EXPANSION FOR WISHINGWELL 
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To illustrate this, we present one example of a complementary pair of 
demon templates. These templates can be instantiated to realize differential 
diagnosis behavior in specific situations. 


A. If the current code segment (or its picture) matches a 
disjunctive subset of the current expectations, select that 
subset. 

B. If no expectation matches the current code segment (or its 
picture), consider a reformulation using the segment's effect as 
a subgoal. 
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9.1. Recapitulatio n 

In this report we have investigated the problem of analyzing elementary 

problem solving protocols. The result of this investigation is the design for 

PAZATN, a domain independent protocol parsing scheme, which was applied to the 

Logo graphics programming domain. Coupled with the Logo ESP's, the design was 

sufficiently well-specified that PAZATN could be hand-simulated for a simple 

example with encouraging results. The foundation for the approach was SPADE, a 

linguistic theory of design in which problem solving is viewed as a structured 

process of planning and debugging. This led us to the definition of an 

interpretation as a parse tree augmented by semantic and pragmatic annotation 
associated with each node. 


A key ingredient in the design is a machine problem solver called PATN. 
PATN employs an augmented transition network to represent fundamental planning 
concepts, including techniques of identification, decomposition, and 


reformulation. Considerable leverage is obtained from PATN's ability to generate 

successively less preferable solution paths, by a series of pragmatically guided 

Planning decisions, as well as from PATN's characterization of certain bugs as 
errors in these planning choices. 

Wu found an analogy to computational linguistics to be fruitful, 
providing insights into data representations and search strategies which are 
characteristic of research in syntactic analysis [Nay 1973; Kaplan 1973] and 
speech recognition (e.g., [Lesser et al. 1975; Paxton & Robinson 1975]). For 
example, the chart representation is used to economically store well-formed 
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substructures. Lookahead, least commitment, and differential diagnosis are 
example strategies used to refine PAZATN's search for a parse, These allow for 
proceeding on the basis of reasonable assumptions when necessary, while retaining 
the ability to modify the interpretation in response to anomalies. 

The analysis procedure has been designed to obtain maximal advantage from 
both top-down guidance from the task description and bottom-up protocol evidence. 
Analysis proceeds by a best first coroutine search of a space of partial 
interpretations. The planchart, a data structure resembling an AND/OR goal tree, 
is used to keep track of expectations. By careful selection of the 
representational scheme, this structure achieves considerable storage economy. 
Partial knowledge of structure and of the status of expectations is recorded 
using a scheme of planchart markings and marker propagations. The planchart is 
incrementally expanded by PATN when existing expectations are inadequate in view 
of the protocol data. A second chart, the datachart, is used to keep track of 
the state of alternative partial interpretations. 

Although PAZATN is not yet a working program, the design is sufficiently 
specific so as to be hand-simulable. In hand-simulation, there is a danger of 
unintentionally drawing upon knowledge which has not been isolated or formalized. 
Care was exercised to avoid this pitfall, and the examples are encouraging 
evidence that the approach is fundamentally sound. Still, hand-simulation is not 

seen as a substitute for implementation. The next phase of the research is to 
implement and experiment with a prototype analyzer. 

PAZATN is a generalization and extension of previous approaches. PAZATN 
grew out of Goldstein's [1974; 1975] plan-finder for MYCROFT. The differences 
are that PAZATN: (a) generates interpretations consistent with the recently 
developed SPADE theory; fW handles the wider range of event types necessary to 
analyze protocols rather than finished programs; fcl takes advantage of the 
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dynamic information in theta additional event types regarding snbgo.l structure 

development, and (d) is not limited to the Logo domain. The SPADE theory 

developed from the MYCROFT theory of program understanding as well as related 

work by Sussman [1973], Papert [1971] and Sacerdoti [1975], [Goldstein 4 

Hiller 1976b] argues that SPADE represents progress over this earlier theorizing. 

PAZATN also complements the features and limitations of analyzers developed at 

Carnegie-Hellon University. The major theoretical advance is a highly structured 

model of program synthesis. The major practical advance is the modularization of 

domain specific knowledge, which indicates that the PAZATN framework ought to be 
applicable to a wide variety of task domains. 

PAZATN is independent of the detailed form of the synthetic formalism: 

it does not intrinsically depend on PATN being an augmented transition network. 

It is only necessary that the synthetic component plan and debug by making a 

series of pragmatic choices which can be summarized by the planchart data 

structure, and that it be capable of generating not one, but an entire space of 

progressively less favored solution paths. Finally, an implicit assumption runs 

throughout the analyzer's design that solutions can be decomposed into syntactic, 

semantic, and pragmatic elements. It may be that any synthetic formalism 

satisfying these constraints is trivially equivalent to an ATN. Such questions 
are notoriously difficult to settle. 

However, an Important issue In the design is the breadth of the synthetic 
theory. There are of course particular omissions such as conditional plans, 
which have been deliberately - but only temporarily - ignored. The greater 
threat comes from the unknown. Even very young children display incredible 
richness in their problem solving. Although SPADE’S origins are partly 
empirical, crucial phenomena - perhaps those most in need of investigation — 
may have been overlooked. This remains a topic for investigation. 
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9.2. Implementation Plans 

There are several ways in which an apparently sound design could fail in 
implementation. The space of partial interpretations could turn out to be very 
large relative to the sources of constraint which have been isolated. The 
variety of knowledge used by human programmers could greatly surpass pur current 
estimates. PAZATN's storage requirements could exceed practical bounds. The 
analyzer could be too rash in its heuristic quest for efficiency, terminating 
prematurely with unacceptable interpretations. Too great a demand could be 
placed on PATN's ability to find reformulations encompassing bottom up evidence. 
Hand-simulations or even partial implementations could overlook such impediments. 

Consequently, complete implementation of a prototype system is essential 
for validating the research. We intend to perform this implementation 
incrementally, beginning with an interactive version. At first, only 
straightforward bookkeeping functions will be automated. The computer will 
record plans and event assignments using the two charts, but decisions regarding 
which interpretations to pursue will be made by a human investigator. This will 
be replaced by a version which performs the routine analysis of most events, only 
requesting help on more difficult cases. Eventually, the analysis will be 
handled completely by the machine. A modeling component for inducing 
personalized ATN's will be implemented, and its predictive power explored. To 
demonstrate PAZATN's understanding of the protocols, a question answering module 
using a formal query language will be constructed to operate over PAZATN's 


output. 
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10. Notes to Book II 


our efforVs t^^it t pA 7 lTu 1 ? 77 L inClU ^ 3 Chapter indicati n3 the direction of 
our efforts to apply PAZATN to the symbolic integration domain. 

mistvninn«i ? ne excepti ° n t0 this is that some irrational errors (such as 
mistypings) can be recognized as unsuccessful manifestations of PATN plans. 

Iln . n 3 ;. This use of demons is inspired by Charniak's [1972] work on story 
nderstanding. Demons are a type of antecedent theorem [Hewitt 1972]. 

4. The chart data structure is due to Kay T19731 and Iranian rio 7 ii 

ouT^c'h LTT gra H hS CL r i . & Sir0Vich 1976 ^ S data^structures'^imilar 3 to 
think 1 tha h t 'hJtlW 8d f 0r l. the basis ® f independent formal considerations. We 

, ,° f the exte nsion from trees to charts as well as the 
incorporation of a larger variety of node types, our plancharts are an 

^ane m ! nt ^ ganeralized 9™*"* along the dimensions of generality? 

storage economy, and expressive power. • 

m n O mn„^ 5 ^ W0 .^ tend t0 provide PAZATN with limited heuristics for recognizing 

wn i , d |H nt K ifi . erS ’ H ° wever ' rel ying on user chosen names for guidance in 
g al would be too unreliable. Hence, to emphasize that such guidance can be 
ispensed with, we assume here that procedure names are unrecognizable. 

and the valne * protoc ?} ev « nt t0 a Planchart leaf, the type of event 

„ n l h a e H v * lue of .MODEL are considered, but the other semantic variables and the 

p . . 9 J" a ^ c assertions are generally not considered. This is a simplification 

For C eximole eS tU 16 possibility for complex semantic and pragmatic ambiguities. 
•Anvirr ^ ’ * 0 interpretations might be identical except for the value of 

elaborated slTahtTvVn h AltboU9b thi * difficulty seems unlikely, PAZATN could be 
elaborated slightly to handle it. Here we ignore the problem and show only the 

Jariah U i r p% tn C H ri ?l 10n ^ th ® name ° f the submodel our diagrams. (The other 

suppressed.) d ^ pragmatic ass ertions, being assumed unambiguous, are 

__ . 7 * Interlisp [Teitelman 1974, pp. 17.10-17.14] provides such a spellina 

corrector. See also [Teitelman 1970]. pemng 

al. 1967, 8 pp SUC 806?807L qUeS ^ ^ COmn ° n US6 ’ ^ ^ eXample ' [Greenblatt et 

tvnp in nr t° r exaaple \ if much more than the typical time elapses between the 
went Ss'LiMatTrf^n :iVe eVen . ts ’ “ is raore Plausible to interpret the second 

assoH th ”, eP1S °i f' m0re specific example involves the frequent 

errors associated with Logo line numbers. There are two such errors: Cl) 

ailing to include a line number when it is needed; and (2) accidentally 

including a line number when it is not needed. Consider the second If murh 

tvne-ilTo/th typical tine ela Pses between the type-in of the line number and the 
yp in of the remainder of the event, it becomes more plausible to interpret the 

£tS.VthL b "? 9y - RUN CVent rather than a la ^ al »»* in«pUc.ble EDIT evenf 

summary stet istYcs" 9 onW^r^’ h0 * ev * r ’ the Preprocessor will accumulate 
summary statistics, only recording the specific data for type-ins which are 
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